Papers with pretrained Transformers
Dense Feature Memory Augmented Transformers for COVID-19 Vaccination Search Classification (2022.emnlp-industry)
Copied to clipboard
Jai Gupta, Yi Tay, Chaitanya Kamath, Vinh Tran, Donald Metzler, Shailesh Bavadekar, Mimi Sun, Evgeniy Gabrilovich
| Challenge: | a new method for classification of COVID-19 vaccination related search queries is proposed . the proposed method uses pretrained Transformers and dense features to generate search insights . |
| Approach: | They propose a machine learning model that detects COVID-19 vaccination related search queries . they use pretrained Transformers to consider dense features as memory tokens that the model can attend to . |
| Outcome: | The proposed model improves the Vaccine Search Insights task by +15% . the proposed model uses pretrained Transformers and traditional dense features . |
Fighting the COVID-19 Infodemic: Modeling the Perspective of Journalists, Fact-Checkers, Social Media Platforms, Policy Makers, and the Society (2021.findings-emnlp)
Copied to clipboard
Firoj Alam, Shaden Shaar, Fahim Dalvi, Hassan Sajjad, Alex Nikolov, Hamdy Mubarak, Giovanni Da San Martino, Ahmed Abdelali, Nadir Durrani, Kareem Darwish, Abdulaziz Al-Homaid, Wajdi Zaghouani, Tommaso Caselli, Gijs Danoe, Friso Stolk, Britt Bruntink, Preslav Nakov
| Challenge: | a dataset of 16K manually annotated tweets is used to analyze disinformation . the democratic nature of social media has raised questions about the quality and the factuality of the information that is shared on these platforms. |
| Approach: | They use a dataset of manually annotated tweets to analyze COVID-19 disinformation . they show that tweets contain fake cures, rumors, conspiracy theories and xenophobia . |
| Outcome: | The proposed dataset shows that it is useful in monolingual vs. multilingual settings. |
Pretrained Transformers Improve Out-of-Distribution Robustness (2020.acl-main)
Copied to clipboard
| Challenge: | Pretrained Transformers are more effective at detecting anomalous or OOD examples, while many previous models are frequently worse than chance. |
| Approach: | They construct a new robustness benchmark with real distribution shifts to measure out-of-distribution generalization for seven NLP datasets and compare them to previous models. |
| Outcome: | The proposed model generalizations for seven datasets show that pretrained Transformers are significantly less effective at detecting anomalous or OOD examples, while many previous models are often worse than chance. |